Information Estimation Functions

This module contains functions for estimating entropy and mutual information between variables using various probabilistic models.

Usage Examples

Estimating Mutual Information

from information_estimation import estimate_information
from models import PixelCNN, GaussianNoiseModel
import numpy as np

measurement_model = PixelCNN(...)
noise_model = GaussianNoiseModel(...)
train_set = np.random.randn(100, 32, 32)  # Example training data
test_set = np.random.randn(100, 32, 32)  # Example test data

mutual_info = estimate_information(measurement_model, noise_model, train_set, test_set)

Running Bootstrapped Estimations

from information_estimation import run_bootstrap
import numpy as np

data = np.random.randn(100, 32, 32)
def estimation_fn(data_sample):
    return np.mean(data_sample)

median, conf_int = run_bootstrap(data, estimation_fn, num_bootstrap_samples=200, confidence_interval=90)

Functions

encoding_information.information_estimation.estimate_information(measurement_model, noise_model, train_set, test_set, confidence_interval=None, num_bootstraps=100, scale_total_mi=False, clean_data=None)

Estimate mutual information in bits per pixel given a probabilistic model of the measurement process p(y) and a probabilistic model of the noise process p(y|x). Optionally, estimate a confidence interval using bootstrapping, which represents the uncertainty in the estimate due to the finite size of the test set.

Parameters:
  • measurement_model (MeasurementModel or List[MeasurementModel]) – A probabilistic model of the measurement process p(y|x) (e.g. PixelCNN, FullGaussian, etc.). If a list, then the model with the lowest NLL (i.e. the best model) is used.

  • noise_model (NoiseModel) – A probabilistic model of the noise process p(y|x) (e.g. GaussianNoiseModel, PoissonNoiseModel).

  • train_set (ndarray, shape (n_samples, ...)) – The training set of noisy measurements, with different shapes depending on data type/model.

  • test_set (ndarray, shape (n_samples, ...)) – The test set of noisy measurements, which should have the same shape as the training set.

  • confidence_interval (float, optional) – Confidence interval for the mutual information, estimated via bootstrapping.

  • num_bootstraps (int, optional) – Number of times to resample the test set to estimate the confidence interval.

  • scale_total_mi (bool, optional) – If True, scales the per-pixel negative log likelihood to total negative log likelihood. Requires AnalyticComplexPixelGaussianNoiseModel.

  • clean_data (ndarray, shape (n_samples, ...)) – Clean data corresponding to the noisy measurements, which may give more accurate estimates of the conditional entropy depending on the noise model. Conventionally this would have the same number of samples as the train_set + test_set combined.

Returns:

  • mutual_info (float) – The mutual information in bits per pixel.

  • (lower_bound, upper_bound) (tuple of floats, optional) – Lower and upper bounds of the confidence interval for the mutual information (if confidence_interval is provided).

encoding_information.information_estimation.analytic_multivariate_gaussian_entropy(cov_matrix)

Compute the entropy of a multivariate Gaussian distribution in a numerically stable manner.

Parameters:

cov_matrix (ndarray, shape (d, d)) – Covariance matrix of the multivariate Gaussian distribution.

Returns:

entropy – The entropy of the multivariate Gaussian in bits.

Return type:

float

encoding_information.information_estimation.nearest_neighbors_entropy_estimate(X, k=3)

Estimate the entropy (in nats) of a distribution using the k-nearest neighbors estimator.

Parameters:
  • X (ndarray, shape (n_samples, n_dimensions)) – The samples from the distribution to estimate the entropy for.

  • k (int, optional) – The number of nearest neighbors to use in the entropy estimate (default is 3).

Returns:

entropy – The estimated entropy in nats.

Return type:

float

encoding_information.information_estimation.estimate_conditional_entropy(images, gaussian_noise_sigma=None)

Estimate the conditional entropy H(Y | X) in nats per pixel, assuming either Gaussian or Poisson noise.

Parameters:
  • images (ndarray) – Clean images, shape (NxHxW) or (NxHxWxC).

  • gaussian_noise_sigma (float, optional) – Standard deviation of the Gaussian noise. If None, Poisson noise is assumed.

Returns:

conditional_entropy – The estimated conditional entropy per pixel.

Return type:

float

encoding_information.information_estimation.run_bootstrap(data, estimation_fn, num_bootstrap_samples=200, confidence_interval=90, seed=1234, return_median=True, upper_bound_confidence_interval=False, verbose=False)

Run a bootstrap estimation procedure using a given estimation function on the provided data.

Parameters:
  • data (ndarray or dict of ndarrays) – The data to use for the bootstrap estimation. If a dictionary, each value must have the same number of samples.

  • estimation_fn (function) – The function to use for estimating the desired quantity. It should accept the data as its input.

  • num_bootstrap_samples (int, optional) – The number of bootstrap samples to generate (default is 200).

  • confidence_interval (float, optional) – The confidence interval for the estimate, expressed as a percentage (default is 90%).

  • seed (int, optional) – Random seed for generating bootstrap samples.

  • return_median (bool, optional) – Whether to return the median (True) or mean (False) of the bootstrap estimates (default is True).

  • upper_bound_confidence_interval (bool, optional) – If True, returns the upper-bound confidence interval (default is False).

  • verbose (bool, optional) – If True, shows a progress bar.

Returns:

  • estimate (float) – The median/mean estimate of the desired quantity across bootstrap samples.

  • confidence_interval (list of float) – The lower and upper bounds of the confidence interval.

encoding_information.information_estimation.estimate_task_specific_mutual_information(noisy_images, labels, test_set_fraction=0.2, patience=None, num_val_samples=None, batch_size=None, max_epochs=None, learning_rate=None, steps_per_epoch=None, num_hidden_channels=None, num_mixture_components=None, do_lr_decay=False, return_entropy_model=False, verbose=False)

DEPRECATED: Use estimate_information() instead.

Estimate the mutual information (in bits per pixel) between noisy images and labels using a PixelCNN entropy model.

Parameters:
  • noisy_images (ndarray) – Noisy images of shape (NxHxW) or (NxHxWxC).

  • labels (ndarray) – Labels of shape (NxK) as one-hot vectors.

  • test_set_fraction (float, optional) – Fraction of the data to be used as the test set for computing the entropy upper bound (default is 0.2).

  • patience (int, optional) – How many iterations to wait for validation loss to improve (used in training). If None, the default is used.

  • num_val_samples (int, optional) – Number of validation samples to use. If None, the default is used.

  • batch_size (int, optional) – Batch size for training the PixelCNN model. If None, the default is used.

  • max_epochs (int, optional) – Maximum number of epochs for training. If None, the default is used.

  • learning_rate (float, optional) – Learning rate for training. If None, the default is used.

  • steps_per_epoch (int, optional) – Number of steps per epoch for training the PixelCNN model.

  • num_hidden_channels (int, optional) – Number of hidden channels in the PixelCNN model.

  • num_mixture_components (int, optional) – Number of mixture components in the PixelCNN output.

  • do_lr_decay (bool, optional) – Whether to decay the learning rate during training (default is False).

  • return_entropy_model (bool, optional) – If True, returns the trained PixelCNN entropy model along with the mutual information (default is False).

  • verbose (bool, optional) – If True, prints out the estimated values during the process.

Returns:

  • mutual_info (float) – The estimated mutual information in bits per pixel.

  • pixelcnn (PixelCNN model, optional) – If return_entropy_model is True, returns the trained PixelCNN model along with the mutual information.

encoding_information.information_estimation.estimate_mutual_information(noisy_images, clean_images=None, entropy_model='gaussian', test_set_fraction=0.1, gaussian_noise_sigma=None, estimate_conditional_from_model_samples=False, patience=None, num_val_samples=None, batch_size=None, max_epochs=None, learning_rate=None, use_iterative_optimization=True, eigenvalue_floor=0.001, gradient_clip=None, momentum=None, analytic_marginal_entropy=False, steps_per_epoch=None, num_hidden_channels=None, num_mixture_components=None, do_lr_decay=False, add_gaussian_training_noise=False, condition_vectors=None, return_entropy_model=False, verbose=False)

DEPRECATED: Use estimate_information() instead.

Estimate the mutual information (in bits per pixel) for a stack of noisy images using a probabilistic model (Gaussian or PixelCNN). Subtracts the conditional entropy assuming Poisson or Gaussian noise. Clean images can be used to estimate the conditional entropy, or it can be approximated from the noisy images themselves.

Parameters:
  • noisy_images (ndarray) – Stack of noisy images or image patches (NxHxW).

  • clean_images (ndarray, optional) – Clean images corresponding to the noisy images. If None, the noisy images themselves are used.

  • entropy_model (str, optional) – The model used for estimating the entropy of the noisy images. Options are: - ‘gaussian’: Uses a stationary Gaussian process model. - ‘pixelcnn’: Uses a PixelCNN model. - ‘full_gaussian’: Uses a full covariance matrix for the Gaussian process.

  • test_set_fraction (float, optional) – Fraction of the data to be used as the test set for computing the entropy upper bound (default is 0.1).

  • gaussian_noise_sigma (float, optional) – If provided, assumes that the noisy images arise from additive Gaussian noise with this standard deviation. Otherwise, Poisson noise is assumed.

  • estimate_conditional_from_model_samples (bool, optional) – If True, estimates the conditional entropy from samples generated by a model fit to the data, rather than the data itself.

  • patience (int, optional) – Number of iterations to wait for validation loss to improve (used for iterative optimization in the Gaussian process model).

  • num_val_samples (int, optional) – Number of validation samples to use (default is None).

  • batch_size (int, optional) – Batch size for training (default is None).

  • max_epochs (int, optional) – Maximum number of epochs for training (default is None).

  • learning_rate (float, optional) – Learning rate for training (default is None).

  • use_iterative_optimization (bool, optional) – If True, performs iterative optimization to refine the stationary Gaussian process estimate (default is True).

  • eigenvalue_floor (float, optional) – Sets the minimum allowed eigenvalue for the covariance matrix in the Gaussian process (default is 1e-3).

  • gradient_clip (float, optional) – If using iterative optimization with a Gaussian model, clip the gradients to this value (default is None).

  • momentum (float, optional) – Momentum for gradient descent in iterative optimization of the Gaussian model (default is None).

  • analytic_marginal_entropy (bool, optional) – If True, uses the analytic entropy of the Gaussian fit for H(Y) instead of upper bounding it with the negative log likelihood.

  • steps_per_epoch (int, optional) – Number of steps per epoch for training the PixelCNN model (if entropy_model is ‘pixelcnn’).

  • num_hidden_channels (int, optional) – Number of hidden channels in the PixelCNN model (if entropy_model is ‘pixelcnn’).

  • num_mixture_components (int, optional) – Number of mixture components in the PixelCNN output (if entropy_model is ‘pixelcnn’).

  • do_lr_decay (bool, optional) – Whether to decay the learning rate during training for the PixelCNN model (default is False).

  • add_gaussian_training_noise (bool, optional) – Whether to add Gaussian noise to the training data instead of uniform noise (default is False).

  • condition_vectors (ndarray, optional) – Conditioning vectors to use for the PixelCNN model (if entropy_model is ‘pixelcnn’), with shape (n_samples, d).

  • return_entropy_model (bool, optional) – If True, returns the trained entropy model along with the mutual information (default is False).

  • verbose (bool, optional) – If True, prints out the estimated values during the process.

Returns:

  • mutual_info (float) – The estimated mutual information in bits per pixel.

  • entropy_model (model, optional) – If return_entropy_model is True, returns the trained entropy model along with the mutual information.